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Shift-and-add is an approach employed to mitigate the phenomenon of resolution 
degradation in images acquired through a turbulent medium. Using this technique, 
a large number of consecutive short exposures is registered below the coherence time 
of the atmosphere or other blurring medium. The acquired images are shifted to 
the position of the brightest speckle and stacked together to obtain high-resolution 
and high signal-to-noise frame. 

In this paper we present a highly efficient method for determination of frames shifts, 
even if in a single frame the object cannot be distinguished from the background 
noise. The technique utilizes our custom genetic algorithm, which iteratively evolves 
a set of image shifts. We used the maximal energy of stacked images as an objective 
function for shifts estimation and validate the efficiency of the method on simulated 
and real images of simple and complex sources. Obtained results confirmed, that our 
proposed method allows for the recovery of spatial distribution of objects even only 
2% brighter than their background. The presented approach extends significantly 
current limits of image reconstruction with the use of shift-and-add method. The 
applications of our algorithm include both the optical and the infrared imaging. 
Our method may be also employed as a digital image stabilizer in extremely low 
light level conditions in professional and consumer applications. 
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Fig. 1. A series of consec¬ 
utive frames, which have to 
be shifted (red arrows) to¬ 
ward the common image cen¬ 
ter (green dashed line) and av¬ 
eraged to produce high resolu¬ 
tion image. 


Introduction 

Images obtained through a turbulent medium (like the atmosphere) suffer from serious quality 
degradation. There are several methods developed to alleviate this problem. One of them is the 
adaptive optics [1], which is a method commonly used in astronomy, microscopy and surveillance. 
In this technique, a high-speed camera monitors continuously the shape of the wave front and the 
closed-loop system compensates for the wave distortions by deforming a special mirror in the optical 
path [2]. However, this approach is very complex, expensive and requires sophisticated equipment 
(e.g. the deformable mirror or the wavefront sensing devices). Moreover, the required wavefront 
diagnostics is available only for the objects much brighter than the background. If there is not 
enough light to perform the wavefront analysis, only the tip-tilt correction is applied [3]. It is 
accomplished by oscillating an additional mirror in optical path, which results in real-time image 
shifting [3,4]. 

Another, much simplified yet efficient approach, is the Lucky Imaging [5], where consecutive short 
exposures are analyzed, shifted to the brightest speckle and stacked together, as depicted in Fig. 1. 
Only the images with the highest Strehl ratio^ are included, hence the final image is the average 
of the frames captured under best atmospheric conditions. The probability of obtaining such good 
quality and unblurred frames is a function of the aperture size of D and the Fried parameter tq [6] : 

P ^ 5.6 exp ^—0.1557 (T)/ro)^^ for D/vq > 3.5. (1) 

However, even those good quality frames are shifted relative to each other. The cancellation of 
tip-tilt component recovers about 50% of spatial distribution in most cases [7,8]. That kind of 
recovery is possible only if the source is intensive enough, so that a single frame contains clearly 
visible brightest speckles. Moreover, the brightest speckle varies very quickly and it is assumed to 
be stable for a very short, so-called coherence time (about 10 ms) [9]. To the best of our knowledge, 
there were no successful attempts to recover the high quality spatial distribution of the objects 
using shift-and-add method, if neither the brightest speckle nor the signal centroid is visible in a 
single exposure. 


^The Strehl ratio is a measure of the quality of optical image formation. It has a value in the range 0 to 1, where 
1 describes an unaberrated optical system. 
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Fig. 2. The consecutive steps of image simulation: a) - simulated counts from the object (Airy 
discs model depicted in the left upper corner), b) - object with added background counts, c) - a 
final simulated image with included Gaussian electronic noise. 


Our method 

In this paper we present a novel technique based on the genetic algorithm (GA), which allows for 
individual frames centring, even if the brightest speckle is not distinguishable, (see example of such 
frame in Fig. 2c). Our method does not require any additional information about observed object, 
so it may be employed in a wide range of applications. 

GA is an iterative technique employed for the optimization purposes in highly nonlinear spaces 
[10-12]. It searches a solution space in order to achieve a given goal, so that in each evolution of 
the algorithm, a part of individuals is randomly modified in hope to produce new, better solutions. 
The best individuals (elite) are transferred directly between the evolutions. Another group in the 
population is created by the mutation of the elite. Finally, in each iteration, a part of individuals 
is created randomly to prevent getting stuck at a local minimum. 

The use of GA on a large number of frames allows to evolve the set of frames shifts toward the 
highest energy of averaged frame, i.e. - to maximize the value of E defined as: 

N M 

« = EE4.' (2) 

x=l y=l 

where Ix^y is the number of counts in a pixel at position (x, y) and A, M denotes the frame’s 
width and height. We used such a fitness function, because only the perfect alignment of all images 
provides the maximum sum of squared counts. As the errors occur in shifts estimations, the counts 
from the object are spread between many pixels in the averaged stack, thus our metric decreases. 
Before the energy calculation, we perform background estimation utilizing a median over all pixels 
in stacked images. Then, the background level is subtracted, hence the influence of background 
counts on the energy measure E is significantly reduced. 

In our simulations, first we used Airy discs as sources [13]. It is the best description of spot light 
that a perfect lens with circular aperture can produce. The disc size is governed by the diffraction 
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limit, what may be observed in spaceborne telescope missions due to a lack of turbulent medium 
between the imager and the object (the interested reader is referred to e.g. [14]). To simulate real 
count results in each pixel, we utilized Poisson distribution, where the mean value was determined 
from the sum of background and object counts [15,16]. 

We present an exemplary result of simulated Airy discs in Fig. 2a, where the objects produce 
61 photons and the entire background counts equals 677. Finally, we combined the counts of the 
background with the ones from the object and added Gaussian readout noise of imager amplifier. 
We adjusted the level of noise and source, so that the object cannot be distinguished from the 
background noise (see Fig. 2c). 

In GA we optimized two variables per each frame - the shifts in X and Y directions. Such an 
approach produces a huge number of variables, thus a whole set of combinations cannot be tested 
in reasonable time even with the most powerful computers. Assuming that the object tip-tilt effect is 
limited to pixels in each direction, we have the following number of possible shift combinations c 
for K images in a series: 

((2p+l)^ + A--l)! 

(2p+l)2! (A--1)! ’ ' ' 

In contrast, with the use of GA, this optimization process can be significantly reduced, since the 
algorithm promotes and modifies only viable shifts combinations while the others are discarded. 
For example, in our experiments, the recovery of 70 frames of size 100x100, took about 30 minutes 
(1500 generations) in the case of simple point-like sources. According to Eq. (3), 3.75E-h479 years 
would be required on the same computer to search a whole combinations space (c= 1.97E-h489 
combinations for iF = 70 images in a series, maximum shift p — 7] the computational time of a 
single shift-and-add procedure is 6 ms). 

We used 700 individuals in our GA population. By an individual we understand the vector of 
shifts in X and Y directions for all images in a series (e.g. for 100 images the individual is 200- 
elements vector). In each iteration, the best solution was transferred directly from the previous 
population and 500 individuals were created by the crossover procedure from the shifts estimations 
of previous 140 individuals (elite). We employed scattered crossover, thus each shift of newly created 
solution was selected from a random individual from the elite. Next 100 individuals were created by 
mutation, where we changed the shifts of previous best individual using the Gaussian distribution 
with a standard deviation equal to 1/3 of maximum available shift. 

The GA final outcomes were the shifts estimations obtained after 1500 iterations. Such value was 
chosen due to the observed satisfactory results in this initial part of the convergence process and 
due to the reasonable computational time (30 minutes). It should be noted, that better outcomes 
may be obtained after much longer GA run. However it would require randomly varying calculation 
time, often reaching tens of hours. 

It is difficult to present strict numeric properties of the method, since each individual run depends 
on its randomly generated starting point. Nevertheless, from multiple tests we observed, that the 
method allows for the recovery of the spatial distribution of objects as dim, as 2% of their mean 
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background. For such intensity of light, about 70 exposures were required to obtain satisfactory 
results using our GA-based algorithm. 

In Fig. 3 we present the outcomes of our algorithm for exemplary images. The first one depicts 
the aforementioned point-like objects. To obtain the remaining two complex images, we cropped 
100 xlOO pixels from the image of a car licence plate and 200x200 pixels of human retina. We 
simulated the required short exposures by reducing significantly the pixels intensities. Then, we 
shifted the frames randomly. We decided to select uniform distribution, so that the estimation was 
more challenging, than it would be for Gaussian distribution, in which a majority of small shift 
deviations is observed. We also accounted for the Poisson and read-out noise of imager amplifier. 
Then, we ran our GA procedures on such a set of frames. 

The presented experiment reflects the imaging process through the turbulent medium, where the 
Fried parameter ro is equal to the lens size. In such a case, the most prominent resolution degra¬ 
dation originates in a global tip-tilt effect within the image plane. 

In our experiments we obtained visually satisfactory results, even if the light flux was about 2-3% 
stronger than the background. The image enhancement is significant and the error maps contain 
only the noise without any visible object signature. The calculations of the signal to background level 
were based on the photon count, measured within green (background only) and blue (background 
and signal) regions (see Fig 3b, g and 1). For the point-like object, the signal-to-background ratio 
was 3.7%. For the complex images it was, respectively 1.78% and 1.92% for the license plate and 
human retina. 

In Fig. 3 we also included the results of frequently used image quality measures: RMSE (Root- 
Mean-Square Error, in counts) and BLUR (see [17] for description). For better readability, BLUR 
outcome was normalized, so that for the reference its value is 0, and for the stack with simulated 
turbulence, it is 1. To reduce the impact of background noise on results of quality measures, all 
images were smoothed using a 3x3 moving average. The reference for the comparisons was obtained 
by stacking images without simulated turbulence. 

In Fig. ?? we present a comparison between calculated and simulated X and Y shifts in an 
exemplary experiment run. The calculated mean shift error was, respectively, 0.13 and -0.2 for 
X and Y direction. The standard deviation was 1.43 (A) and 1.08 (T). As predicted, for the 
most of the frames, the shifts estimations provided by our method were the same or very close to 
the reference ones. 


Summary 

In this paper we demonstrated the first results of spatial distribution recovery in shift-and-add 
method when no brightest speckle nor the signal centroid is visible in a frame. The spatial distribu¬ 
tion, to the best of our knowledge, not possible to resolve by any other approach, was successfully 
reconstructed. The proposed GA method may be employed in a wide range of applications includ¬ 
ing astronomy, medicine or surveillance. Moreover, it is an automatic method and no knowledge 
about spatial distribution, number of sources or their position is required. It is also very easy to 
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Fig. 3. The results of image enhancement employing our method: a), f), k) - exemplary single 
frames, the contrast in these images was increased for better readability; b), g), 1) - average of 70 
frames without simulated turbulence, green and blue rectangles represent the areas utilized for the 
signal-to-background measurements; c), h), m) - average of 70 frames with simulated turbulence; 

d) , i), n) - average of 70 frames after applying image shifts corrections provided by our algorithm; 

e) , j), o) - the difference between the reference image and the GA enhanced result (error map). 
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Real displacement vs. displacement recovered by GA 

Fig. 4. Shift canceling 
displacement computed 
by GA vs. real dis¬ 
placement - the results 
of an exemplary ex¬ 
periment run performed 
on 70 frames of point¬ 
like objects depicted in 
Fig. 2. 
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Real displacement [pix] 

implement, as GA solvers are presently available in numerous software packages. 

The proposed technique is suitable for real and simulated data in both optical and infrared bands. 
Our method may be also employed as a digital image stabilizer in extremely low light level conditions 
in both professional and consumer applications. 
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